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Formative, in-process evaluation results can be used 
to refine or "fine tune" Even Start programs. The assumption is that 
the programs have obtained comprehensive and appropriate empirical 
data on the assessment of the program as a whole as well as its 
specific goals. This six-step model uses evaluation data to improve 
program delivery. The steps are as follows: (1) convene a planning 
team of five to eight people who represent various aspects of Even 
Start delivery; (2) develop a shared understanding about what the 
local evaluation results mean for or about the program; (3) explore 
reasons for the results (the problems); (4) generate and evaluate 
possible solutions; (5) develop and implement the plan for program 
refinement; and (6) evaluate the success of the plan. Following the 
presentation and explanation of this model, two fictional examples of 
the model in action are provided. One describes a program that uses 
that model to solve the program's retention problems. The second 
focuses on a program that encounters unexpected problems with Even 
Start parents' reading ability. (YLB) 



PUB DATE 
NOTE 
PUB TYPE 

EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 
ABSTRACT 



A A A A A A A * A * A ft A A A ft ft * * A A ft ft ft ft * ft * * A ft * A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A • 

* Reproductions supplied by EDRS are the best that can te made 

* from the original document. 

A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A 



Using Evaluation Results 
To Refine Family Literacy Programs 



Occasional Paper #5 

Nancy Padak 
Tim Rasinski 

Kent State University 
June, 1994 



US DEPARTMENT OF EDUCATION 
o-'r. >■ o^i' .1,(1 1'rivr.^r"; "PERMISSION TO REPRODUCE THIS 



EDUCATIONAL RESOURCES INFORMATION 



MATERIAL HAS BEEN GRANTED BY 



CENTER (ERIC) 



r^This document has been reproduced as 
" received from the person or organization 





• Points ol view 01 opinions stated m this 
document do nol necessanly represent 
oltinat OERI position or policy 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC).'" 



This work was su; 
Federal Assistan 
#062976-EV-SD-94 




i TEST COPY AVAILABLE 



/ 



Using Evaluation Results 
To Refine Family Literacy Programs 

Occasional Paper #5 

Nancy Padak 
Tim Rasinski 

Kent State University 

June, 1994 

When the "E" word-- evaluation-- is mentioned, reac- 
tions from most Even Start professionals range from weary 
sighs and rolling eyes to vocal groans and visible 
frowns. Evaluation activities are viewed as extra work, 
in many cases work that prevents delivery of the instruc- 
tional program itself. After all, testing isn't teach- 
ing. 

Evaluation will always be part cf projects like Even 
Start, however. The projects are operated with public 
funds, and the public has the right to know about the 
effect of those funds on participating families. Other 
family literacy practitioners are also interested in ES 
evaluation data. These professionals may wish to know 
"what works" under what circumstances, so that they may 
learn from others' experiences and avoid ethers' diffi- 
culties. In fact, Even Start's designation as a demon- 
st ration project requires this sort of sharing. National 
evaluation projects are designed to provide both of these 
types of information about ES projects throughout the 
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nation. 

But the "broad strokes" picture painted by national 
evaluation data may not be very helpful to individual 
programs. This is why local evaluations are also criti- 
cal. The purpose of local evaluations is to determine 
the extent to which individual programs are meeting their 
goals, both (a) the "external" objectives developed for 
program applications, which respond to ES legislation, 
and (b) the "internal" objectives that reflect more 
closely what the program really hopes to accomplish. If 
local evaluations are developed and conducted carefully, 
they can provide formative results, or in-process guid- 
ance for improving all aspects of an Even Start program, 
as well as sutnmative results or final evaluations of 
program effectiveness. 

Our focus in this paper is on the use of formative, in- 
process evaluation results to refine or "fine tune" 
programs. We assume that the programs have obtained 
.comprehensive and appropriate empirical data on the 
assessment of the program as a whole as well as its 
specific goals. First, we present and explain a six-step 
model for using evaluation data to improve program deliv- 
ery. Then we provide two examples of the model in ac- 
tion. Although the examples we present relate directly 
to ES, the model itself is useful for any type of pro- 
gram. 



The Model 
Step 1 : Convene a Planning Team 

ES programs wishing to use local evaluation results to 
refine their programs can do so. We recommend that you 
convene a small planning team (5-8 people) , whose members 
represent various aspects of ES delivery, to assist you. 
This group should include parent representation and may 
include members who are not routinely involved in ES. 
For example, a public school curriculum specialist or a 
college or university educator might be a helpful addi- 
tion to the planning team, depending on the nature of the 
problems the group is exploring. Together, members of 
this planning team should follow the steps outlined below 
to ensure a systematic and thorough consideration of both 
the local evaluation results and the direction they 
provide for improving program delivery. 
Step 2 : Understand the Results 

The second step involves developing a shared under- 
standing about what the local evaluation results mean for 
or about your ES program. In other words, the planning 
group will need to decide if the results "make sense." 
This ordinarily involves thinking about and evaluating 
the measures used to obtain the results --Do they make 
sense or "ring true" from the perspective of program 
part* icipants? 

3 
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One way to appraise results of a local evaluation is to 
assess the validity (sometimes called "truthfulness") of 
the measures used to generate those results. To do this, 
the planning group should read the evaluation report 
carefully, find the measures used to evaluate the goals 
and that led to the identification of the problem, and 
evaluate the practical worth of those measures . To do 
so, the group should ask, Are these good and reasonable 
measures of the level of achievement for this goal? For 
example, "successful retention" might be defined as 
participation for one month or one year. Likewise, 
adults' writing ability might be assessed using standard- 
ized usage tests or evaluation of pieces of their writing 
that were completed as part of ES activities. Better 
correlation between the definition and the measures se- 
lected for use in the evaluation results in greater 
validity or truthfulness of the results. 

This assessment of validity or truthfulness can yield 
three possible outcomes. If the planning group decides 
that the measures are valid, then they should proceed to 
step three of the model, because they have decided that 
the problem(s) identified in the evaluation report is 
real. If the planning group decides that the measures 
are invalid, then the ES coordinator and local 
evaluator(s) should meet to refine the evaluation plan so 
that subsequent assessments can yield more useful infor- 
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mation. And finally, if the planning group has questions 
about the validity of the measures, they can seek addi- 
tional information, perhaps by asking the local evalua- 
tors to provide other perspectives about the aspect of 
the program under question. For example, the planning 
group might request that another type of measure be added 
to subsequent evaluations of the issue under question, or 
they might ask local evaluators if already-collected data 
could provide another view of the issue under study. 
Step 3 : Explore Reasons for the Results 

If completing the second step in the model convinces 
the planning group that there is indeed a problem with 
some aspect of ES delivery, the next step is to try to 
identify reasons for the problem's existence. Most 
problems associated with complex delivery systems like ES 
are themselves complex, so this will likely be no easy 
task. Nevertheless, it's important to try to identify 
reasons because they can provide focus for developing 
solutions . 

We recommend that the search for reasons be as objec- 
tive as possible. Try to avoid simplistic, "blame the 
victim" reasons. For example, if retention of ES fami- 
lies is identified as a problem, it might be tempting to 
conclude that parents "just don't care enough" to attend 
regularly. Even if this is true, however, it is likely 
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that particular aspects of the program contribute to this 
attitude-- program location or times may be inconvenient, 
for example, or adults may not view ES activities as 
interesting or helpful. A "blame the victim" reason 
cannot yield program refinements. Rather, the planning 
team should attempt to generate possible reasons that can 
result in program improvement somehow. To do otherwise 
renders the entire evaluation process fruitless. 

We also recommend that the search for reasons be con- 
ducted broadly (i.e., by considering opinions of the 
planning group, those of other ES staff and participants, 
and, if warranted, those of outside experts) . Brief 
person-to-person interviews or conversations may be an 
efficient way to gather this information. In the case of 
problems with retention, for example, each member of the 
planning group might agree to talk with three other 
persons about why retention might be a problem in the ES 
program. When the planning group reconvenes, a group 
list of possible reasons can be developed, and items on 
the list can be evaluated as "likely" or "not likely." 

Depending upon the nature of the problem, it may also 
be necessary to observe instruction, analyze instruction- 
al materials, or interview learners. Teachers may ini- 
tially feel a bit insecure about classroom observations, 
but anxiety will be lessened if teachers are represented 
on the planning team and involved in the solution of 
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identified problems. After all, everyone involved in ES 
shares the desire to develop a program that is as 
succpssful as possible in meeting its goals. 
Step 4 : Generate and Evaluate Possible Solutions 

The time and energy devoted to the two preceding 
steps in the model can help you focus on likely causes 
for problems. Such a focus is necessary, of course, 
before solutions can be generated. The fourth stage in 
the model is to do just that-- generate possible solu- 
tions and then select the one most likely to solve the 
problem. 

Brainstorming is probably the most effective way to 
accomplish this step. We recommend that the brainstorm- 
ing be conducted in two phases. During the first, the 
goal is for the planning group to list as many solutions 
to the problem as possible. Consequently, initial direc- 
tions should remind them not to evaluate or elaborate 
upon particular suggestions. Rather, generating lots of 
ideas is the goal. As ideas are suggested, record them 
on chart paper or a chalkboard. 

This initial brainstorming will probably take no more 
than 10 or 15 minutes. Be patient after the first lull 
in conversation, however; it's been our experience tha*~ 
brainstorming suggestions often come in "waves." What's 
on the tips of our tongues is offered first. After these 



ideas are shared, we need a bit of time to think about 
others, and often these "second wave" ideas are more 
complex and thoughtful . 

After possible solutions have been generated, the group 
should evaluate them. The goal of this second phase is 
to select the one solution that the group believes is 
most likely to solve, the problem. in order to accomplish 
this, the group will need to think about qualities of 
effective solutions. Such qualities as feasibility given 
program resources and constraints and allowable activi- 
ties given federal regulations will probably need to be 
considered. It might be helpful to list the qualities 
that the group generates for continued reference. 

Next, individual members of the brainstorming group 
assess the possible solutions. The easiest way to accom- 
plish this is probably to create a "short list" for 
further consideration: a) ask each member of the group 
to select no more than three likely solutions in rank 
order; b) poll the group, recording the number of first 
place votes, second place votes, and third place votes 
for each possible solution; and c) examine the results 
to see if the group agrees on the two or three most 
likely solutions. 

Solutions on the "short list" should be discussed 
thoroughly so that one can be selected for implementa- 
tion. Discussions can be left unstructured or can focus 
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on creating lists of advantages and disadvantages of 
particular choices. This discussion may reveal refine- 
ments that can strengthen particular solutions. The 
group should strive for achieving unanimous support for 
one solution. 

Step 5_l Develop and Implement the Plan for Program Re- 
finement 

Having proceeded this far,, you know where you want the 
ES program to go, and you have group consensus about this 
direction. The next stage of the model is designed to 
help you figure out how to get there. To do this, you 
will need to develop a strategic plan, based on your 
chosen solution, that lists: a) all activities that need 
to be accomplished in the order in which they must occur, 
b) the resources (e.g., financial, human) needed to 
accomplish each activity, c) a timeline for accomplishing 
each activity, and d) the person or people responsible 
for oversight or the actual conduct of each activity. 

Brainstorming can also be effective for creating a 
comprehensive list of activities, which will then need to 
be organized in some way. Depending upon the nature of 
the solution, activities may need to be organized chrono- 
logically, in logical progression, or, if resources are 
limited, in priority order. A good list of activities 
will provide a step-by-step blueprint for moving the 
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program from where it is to where it wants to be. Toward 
that end, several people should probably review the 
proposed activity list for completeness . After the 
activity list is developed, decisions can be made about 
other aspects of the plan, such as resources, timelines, 
and people responsible. 

A chart for summarizing decisions related to the 
strategic plan is appended to this paper. We recommend 
that the final version of the plan be cast into chart 
form and shared with all those interested in or affected 
by it. Moreover, we recommend that the ES coordinator 
periodically seek input from the persons responsible for 
particular activities to ascertain that progress is being 
made in a timely manner. 

Step 6_l Evaluate the Success of the Plan 

If local evaluators have been involved in previous 
steps of the process, then they can probably assist in 
the development of a scheme to determine the impact of 
the program refinement. If local evaluators have not 
been previously involved, now is the time to contact 
them. Share with them the plan for program refinement 
and the particular aspects of your ES program that you 
hope the refinement will affect. Ask them to be particu- 
larly careful with the next local evaluation of those 
particular aspects. Ask them to include a separate 
section about the success of the refinements in their 
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next evaluation report. 

Examples of the Model in Action 
Below we provide fictional accounts of how two ES pro- 
grams used this model to refine their programs based on 
the results of their local evaluations. We hope that 
these concrete examples will help you see how the model 
can be employee! in your own program. 
The Problem: Retention 

It didn't take Jane Smith, ES coordinator, long to 
understand the most significant finding in her program's 
local evaluation: They had retention problems. She knew 
this was the case, and program records documented it. 
Moreover, she knew that several staff members were con- 
cerned about retention. Armed with evaluation results 
pointing to this problem and knowing she had some staff 
support for attempting to solve it, Jane decided to 
convene a group to explore the problem more thoroughly 
and to generate some solutions. She invited the adult 
education teacher, the parenting education teacher, two 
ES parents, and the local adult education director to 
participate with her. 

The planning group first attempted to judge the severi- 
ty of their retention problem by referring to printed 
accounts of retention in other adult education programs 
and by exploring community factors that might influence 

11 



ERLC 



13 



retention in their ES program. They learned that reten- 
tion is a common problem in adult education programs, but 
that family literacy programs generally have higher 
retention rates than other types of adult education 
programs. They also learned that several neighborhood 
schools in their community also had retention problems; 
some had up to 75% student turnover each year. 

Seeking information from published literature and from 
their local community helped the planning group put the 
ES retention problem in perspective. However, the group 
decided to explore the issue further by interviewing 
several parents who had either persisted in the program 
or dropped out of ES but stayed in the community. Their 
questions to parents were simple: Why did you choose to 
leave the program? and How could the ES program better 
meet your needs? 

Although parents offered many suggestions for program 
improvement, one common complaint seemed directly related 
to the retention problem: ES dropouts did not see the 
value of ES instruction, especially the adult education 
component. Persisters, too, seemed unsure about the 
adult education component. Although they stayed with ES, 
they said it was to benefit their children. The adult 
educator, who was part of the planning group, indicated 
that his curriculum was conventional, workbook- driven and 
skills-oriented. He also indicated that he wished to 
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alter his curriculum but didn't know how to begin. 

After discussing and listing a variety of possible 
solutions, from providing tangible incentives to provid- 
ing periodic followup to students, the solution to this 
problem seemed obvious, although the group recognized 
that time, energy, and support would be necessary to 
solve it. For the short term, the planning group recom- 
mended that the adult educator talk with learners about 
the value of their instructional sessions and involve 
them, whenever possible, in establishing academic goals. 
Furthermore, the planning group recommended staff devel- 
opment designed to help all ES staff develop and imple- 
ment learner-centered curriculum as a long-term solution 
to the retention problem. Specifically, their advice was 
to focus on the interrelationship of program components 
so that the entire ES effort could better meet learners' 
needs and interests. 

Jane was a bit surprised that the best solution to 
fostering retention appeared to be in the classroom. But 
she trusted that the planning group had considered the 
issue carefully, and she looked forward to the next local 
evaluation so that the group could see the results of 
their efforts. 

The Problem: Adults 1 Reading Ability 

Unlike Jane, Bob Brown, another ES coordinator, was 
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shocked when he read an evaluation report for his program 
that indicated problems with ES parents 1 reading ability. 
He hadn't predicted problems in this area; the two ES 
adult educators were also surprised when Bob shared 
evaluation results with them. The three decided to 
pursue the issue by creating a planning group that also 
included a parent representative and the county reading 
supervisor . 

When the planning group read the evaluation report 
carefully, they discovered a potential explanation for 
the discouraging results. The reading evaluation was 
based on results of a standardized comprehension test, 
which consisted of having adults read short paragraphs 
and answer multiple choice questions about them in a 
timed situation. The county reading supervisor helped 
the group see the limited definition of reading that 
provided the basis for the test. Moreover, the adult 
educators recalled that several parents seemed very 
anxious on the day they had administered the test. As a 
result of this discussion, the planning group decided 
that they questioned the validity of the standardized 
test as an accurate and comprehensive measure of a per- 
son's reading ability. For this reason, they determined 
that it was unnecessary to pursue possible program re- 
finements based on these questionable evaluation results. 
Instead, they recommended that Bob and the reading super- 
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visor work with the local evaluators to refine the ways 
in which reading ability would be assessed in the ES 
program. 

Conclusion 

No matter how effective a particular family literacy 
program, refinements are always possible. Systematic and 
comprehensive local evaluation should provide direction 
for these program refinements. The five-step model that 
we describe in this paper can help ES professionals 
evaluate potential problems that are identified in local 
evaluation reports, understand their causes, and develop 
effective plans for solving them. 

We know this model can work. We have used it in many 
situations to help educators solve program- related prob- 
lems and to refine program delivery. As family literacy 
professionals, we share the hope that our programs will 
assist families in improving the quality of their lives. 
Periodic and systematic attention to program refinement 
can increase the likelihood that all of our goals will be 
met . 
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